Users, Queries, and Bad Abandonment in Web Search

After a user submits a query and receives a list of search results, the user may abandon their query without clicking on any of the search results. A bad query abandonment is when a searcher abandons the SERP because they were dissatisfied with the quality of the search results, often making the user reformulate their query in the hope of receiving better search results. As we move closer to understanding when and what causes a user to abandon their query under different qualities of search results, we move forward in an overall understanding of user behavior with search engines. In this thesis, we describe three user studies to investigate bad query abandonment. First, we report on a study to investigate the rate and time at which users abandon their queries at different levels of search quality. We had users search for answers to questions, but showed users manipulated SERPs that contain one relevant document placed at different ranks. We show that as the quality of search results decreases, the probability of abandonment increases, and that users quickly decide to abandon their queries. Users make their decisions fast, but not all users are the same. We show that there appear to be two types of users that behave differently, with one group more likely to abandon their query and are quicker in finding answers than the group less likely to abandon their query. Second, we describe an eye-tracking experiment that focuses on understanding possible causes of users' willingness to examine SERPs and what motivates users to continue or discontinue their examination. Using eye-tracking data, we found that a user deciding to abandon a query is best understood by the user's examination pattern not including a relevant search result. If a user sees a relevant result, they are very likely to click it. However, users' examination of results are different and may be influenced by other factors. The key factors we found are the rank of search results, the user type, and the query quality. For example, we show that regardless of where the relevant document is placed in the SERP, the type of query submitted affects examination, and if a user enters an ambiguous query, they are likely to examine fewer results. Third, we show how the nature of non-relevant material affects users' willingness to further explore a ranked list of search results. We constructed and showed participants manipulated SERPs with different types of non-relevant documents. We found that user examination of search results and time to query abandonment is influenced by the coherence and type of non-relevant documents included in the SERP. For SERPs coherent on off-topic results, users spend the least amount of time before abandoning and are less likely to request to view more results. The time they spend increases as the SERP quality improves, and users are more likely to request to view more results when the SERP contains diversified non-relevant results on multiple subtopics

Learning Factors and Determining Document-level Satisfaction In Search-as-Learning

An important use of search engines is as a tool for learning. Search engines help users find learning material and increases their knowledge in various topics. The underlying process of learning while web searching and which documents a search engine should return to enhance the learner's comprehension and learning is a new area of research. In order to build better search engines to supplement the learning process and overall satisfaction, documents the learner searches for should be investigated. In this thesis, we propose six different factors that may be associated with learning and show which are significant in determining document-level satisfaction. We describe a lab-based user study in which each participant was assigned to a learning task with a pre and post quiz to measure their increase in knowledge after reading the selected documents. Using data collected at different stages of the study, our results indicate that documents with broadness of content, as well as novelty of information, are significant in determining satisfaction. We also show qualitative results that indicate a broader to more specific ordering of documents content is preferred for easier processing and retention of information. Our study provides insight into the characteristics of documents learners prefer to read and the order these documents should be presented to the learner, and provides us a better understanding of the learning process that occurs during search-as-learning related tasks

Patterns of Search Result Examination: Query to First Action.

To determine key factors that affect a user's behavior with search results, we conducted a controlled eye-tracking study of users completing search tasks using both desktop and mobile devices. We focus our investigation on users' behavior from their query to the first action they take with the search engine results page (SERP): either a click on a search result or a reformulation of their query. We found that a user deciding to reformulate a query rather than click on a result is best understood as being caused by the user's examination pattern not including a relevant search result. If a user sees a relevant result, they are very likely to click it. Of note, users do not look at all search results and their examination may be influenced by other factors. The key factors we found to explain a user's examination pattern are: the rank of search results, the user type, and the query quality. While existing research has identified rank and user types as important factors affecting examination patterns, to our knowledge, query quality is a new discovery. We found that user queries can be understood as either of weak or strong quality. Weak queries are those that the user may believe are more likely to fail compared to a strong query, and as a result, we find that users modify their examination patterns to view fewer documents when they issue a weak query, i.e. they give up sooner.Natural Sciences and Engineering Research Council of Canada, Grant CRDPJ 468812-14 || Natural Sciences and Engineering Research Council of Canada, Grant RGPIN-2014-03642 || Google || University of Waterlo

A Study of Immediate Requery Behavior in Search

© Haotian Zhang, Mustafa Abualsaud and Mark D. Smucker, 2018. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in the Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, (CHIIR'18), ACM. , https://doi.org/10.1145/3176349.3176400When search results fail to satisfy users' information needs, users often reformulate their search query in the hopes of receiving better results. In many cases, users immediately requery without clicking on any search results. In this paper, we report on a user study designed to investigate the rate at which users immediately reformulate at different levels of search quality. We had users search for answers to questions as we manipulated the placement of the only relevant document in a ranked list of search results. We show that as the quality of search results decreases, the probability of immediately requerying increases. We find that users can quickly decide to immediately reformulate, and the time to immediately reformulate appears to be independent of the quality of the search results.Finally, we show that there appears to be two types of users. One group has a high probability of immediately reformulating and the other is unlikely to immediately reformulate unless no relevant documents can be found in the search results. While requerying takes time, it is the group of users who are more likely to immediately requery that are able to able find answers to questions the fastest.Natural Sciences and Engineering Research Council of Canada (Grants CRDPJ 468812-14 and RGPIN-2014-03642), in part by Google, and in part by the University of Waterloo

Learning Trustworthy Web Sources to Derive Correct Answers and Reduce Health Misinformation in Search

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. SIGIR ’22, July 11–15, 2022, Madrid, Spain © 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-8732-3/22/07...$15.00 https://doi.org/10.1145/3477495.3531812When searching the web for answers to health questions, people can make incorrect decisions that have a negative effect on their lives if the search results contain misinformation. To reduce health misinformation in search results, we need to be able to detect documents with correct answers and promote them over documents containing misinformation. Determining the correct answer has been a difficult hurdle to overcome for participants in the TREC Health Misinformation Track. In the 2021 track, automatic runs were not allowed to use the known answer to a topic’s health question, and as a result, the top automatic run had a compatibility-difference score of 0.043 while the top manual run, which used the known answer, had a score of 0.259. The compatibility-difference measures the ability of methods to rank correct and credible documents before incorrect and non-credible documents. By using an existing set of health questions and their known answers, we show it is possible to learn which web hosts are trustworthy, from which we can predict the correct answers to the 2021 health questions with an accuracy of 76%. Using our predicted answers, we can promote documents that we predict contain this answer and achieve a compatibility-difference score of 0.129, which is a three-fold increase in performance over the best previous automatic method.Natural Sciences and Engineering Research Council of Canada, RGPIN-2020-04665, RGPAS- 2020-00080 || Mitacs || Compute Canada || University of Waterloo

APS: An Active PubMed Search System for Technology Assisted Reviews

Systematic reviews constitute the cornerstone of Evidence-based Medicine. They can provide guidance to medical policy-making by synthesizing all available studies regarding a certain topic. However, conducting systematic reviews has become a laborious and time-consuming task due to the large amount and rapid growth of published literature. The TAR approaches aim to accelerate the screening stage of systematic reviews by combining machine learning algorithms and human relevance feedback. In this work, we built an online active search system for systematic reviews, named APS, by applying an state-of-the-art TAR approach-Continuous Active Learning. The system is built on the top of the PubMed collection, which is a widely used database of biomedical literature. It allows users to conduct the abstract screening for systematic reviews. We demonstrate the effectiveness and robustness of the APS in detecting relevant literature and reducing workload for systematic reviews using the CLEF TAR 2017 benchmark

External validation and recalibration of an incidental meningioma prognostic model - IMPACT: protocol for an international multicentre retrospective cohort study

Introduction: Due to the increased use of CT and MRI, the prevalence of incidental findings on brain scans is increasing. Meningioma, the most common primary brain tumour, is a frequently encountered incidental finding, with an estimated prevalence of 3/1000. The management of incidental meningioma varies widely with active clinical-radiological monitoring being the most accepted method by clinicians. Duration of monitoring and time intervals for assessment, however, are not well defined. To this end, we have recently developed a statistical model of progression risk based on single-centre retrospective data. The model Incidental Meningioma: Prognostic Analysis Using Patient Comorbidity and MRI Tests (IMPACT) employs baseline clinical and imaging features to categorise the patient with an incidental meningioma into one of three risk groups: low, medium and high risk with a proposed active monitoring strategy based on the risk and temporal trajectory of progression, accounting for actuarial life expectancy. The primary aim of this study is to assess the external validity of this model. Methods and analysis: IMPACT is a retrospective multicentre study which will aim to include 1500 patients with an incidental intracranial meningioma, powered to detect a 10% progression risk. Adult patients ≥16 years diagnosed with an incidental meningioma between 1 January 2009 and 31 December 2010 will be included. Clinical and radiological data will be collected longitudinally until the patient reaches one of the study endpoints: intervention (surgery, stereotactic radiosurgery or fractionated radiotherapy), mortality or last date of follow-up. Data will be uploaded to an online Research Electronic Data Capture database with no unique identifiers. External validity of IMPACT will be tested using established statistical methods. Ethics and dissemination: Local institutional approval at each participating centre will be required. Results of the study will be reported through peer-reviewed articles and conferences and disseminated to participating centres, patients and the public using social media